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ABSTRACT 



Generalizations of two-stack-sortable permutations 

A dissertation presented to the faculty of 
the Graduate School of Arts and Sciences of 
Brandeis University, Waltham, Massachusetts 

by Dapeng Xu 

In this thesis, we apply the stack sorting operator to r-permutations and construct 
the functional equation for the generating function of two-stack-sortable /c-tuple r- 
permutations counted by descents by using a factorization similar to Zeilberger's. We 
solve the functional equation and give explicit formulas for the number of two-stack- 
sortable r-permutations. 
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CHAPTER 1 



Introduction 



The operation of stack sorting was first studied by Knuth [T^, p. 239]. He de 



scribed the operation by a raihoad cars swapping algorithm. Later West described 



this operation in terms of a simple card game . 

The operation uses a stack and can be described as follows. Let vr be a word with 
distinct letters in the alphabet {1, 2, 3, . . . }. We call vr the input of the operation. At 
the ith step, we compare the ith letter a of the word tt with the top letter on the stack 
(if any). If a is smaller or the stack is empty, we put a on the top of the stack. If a is 
bigger, we move the top letter on the stack to the output and then again compare a 
with the new top letter on the stack (if any). We repeat this until we put a onto the 
stack. When the input is empty, we move all the letters on the stack (if any) to the 
output in order from top to bottom. 



West also gave a recursive definition of the stack sorting operation. |27, 26 . 



Definition 1.0.1. Let vr = aia2 ■ ■ ■ a„ be a word in the alphabet P = {1,2,...}, 
having all its letters distinct. If = 0, let 5'(7r) be the empty word. Otherwise, let 
S{7t) be obtained by permuting the letters of vr as follows: if m = max(ai, 02, ... , a^) 
and n = irimnr, then 

^(Tr) = S{ni)S{nr)m. 

It is easy to see that this recursive definition of the stack sorting operation is 
equivalent to the description in terms of stacks. For a given word vr = aia2 ■ ■ - an in 
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the alphabet P = {1,2,...}, having all its letters distinct, we can always write it as 
TT = TTimTTr where m = max(ai, 02, ... , dn)- According to the description, since m is 
bigger than any letter in tti, all the letters in tti will be moved to the output before we 
can put m onto the stack. Also since m is bigger than any letter in vr^, m will not be 
removed from the stack until all the letters in vr^ go through the stack. That is exactly 
what the recursive definition says. 

The problem of stack sorting was then generalized and reseached in a number of 
ways. Among these, the problems of enumerating t-stack-sortable permutations inter- 
est most people. We say that a permutation vr of [n] is a t-stack-sortable permutation if 
^^(Tr) is the identity permutation, and that vr is exactly t-stack-sortable if vr is t-stack- 
sortable but not (t — l)-stack-sortable. Knuth |I5| , p. 239] proved that the number of 
1-stack-sortable permutations of [n] is the Catalan number C„ = (^^)/(^ + !)• West 
p6| , proved that all permutations of [n] are {n — l)-stack-sortable, that {n — 2)\ are 
exactly (n — l)-stack-sortable, and that |(n — 2)! + (n — 3)! are exactly (n — 2)-stack- 
sortable. 

Characterizations of 1-stack-sortable permutations and 2-stack-sortable permuta- 
tions were also given using the following notion of pattern avoidance. Let vri = 
01^2 ■ ■ ■ a„ be a permutation of [n] and let tt2 = &1&2 ■ ■ ■ be a permutation of [k]. We 
say that tti contains a type tt2 subsequence if there exists 1 < < ^{,2 < ' " ' < ^ 
such that < < ■ ■ ■ < aj^, . We say tti avoids n2 if tti contains no 7r2-subsequence. 
For example, a permutation vr avoids 231 if it cannot be written as ■ ■ ■ a ■ ■ -b ■ ■ ■ c- ■ ■ 
so that c < a < b. 

Tarjan [^] proved that a permutation is 1-stack-sortable if and only if it avoids 
the pattern 231. West ||26|, ^ proved that a permutation fails to be 2-stack-sortable 
if it contains a subsequence of type 2341 or a subsequence of type 3241 which is not 
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part of a subsequence of type 35241 and it is two-stack-sortable if it contains no such 
subsequence. West also conjectured the number of 2-stack-sortable permutations of 
[n] to be m 

2(3n)! 
(n + 1)! (2n+ 1)!' 

The conjecture was first proved by D. Zeilberger |2^. Later, two bijections between 



two-stack-sortable permutations and non-separable planar graphs were given by S. 
Dulucq, S. Gire and O. Guibert 0; I. P. Goulden and J. West |jlT|]. More contributions 
to this problem were given by Miklos Bona, Mireille Bousquet-Melou, Leopold Travis 
and others g |, |, |, |, H . 



1.1. The generalization 

In this thesis, we generalize the ordinary permutations to r-permutations \T3, T^, 



161 , p77| , pTq] and then enumerate the number of 2-stack-sortable r-permutations under 



the generalized stack-sorting operation on r-permutations. 

Definition 1.1.1. If a permutation aia2 ■ ■ ■ am of {l*", 2'", . . . , n"^} satisfies the con- 
dition that ii i < j < k and then ttj < ai, we call it an r -permutation of 
[n]. 

Definition 1.1.2. Let vr be an r-permutation of [n]. For n > 0, we can write vr as 
TT = aina2n ■ ■ ■ nar+i- The stack-sorting operation S on r-permutations is defined by 

^(Tr) = S{ai)S{a2) ■ ■ ■ S{ar+i)n. 

When n = 0, i.e., when vr is an empty permutation, we define 5'(7r) to be 0. Note that 
^(Tr) is an ordinary permutation. 
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Definition 1.1.3. Given an r-permutation tt and a letter a in tt, we call a a type 
i descent (i = 1, . . . , r) if the ith occurrence of a is immediately followed by a smaller 
letter, and we call a a type descent if the first occurrence of a immediately follows 
a smaller letter. We denote the set of type i descents of an r-permutation tt by tt*^*^ 
(z = 0,...,r). 

Remark 1.1.4. Notice that the r-permutations are usually defined the other way, 
i.e., a permutation a\a2 ■ ■ ■ am of {1^, 2'', . . . , rf} is an r-permutation if it satisfies the 
condition that ii i < j < k and = a^, then > a^. The reason for the change 
is that we expect S'(7r) to be an ordinary permutation for an r-permutation tt, while 
the traditional definition of r-permutation can not give us that. For example, 123321 
is a 2-permutation under the traditional definition, but 5'(123321) = 121233. This 
behavior is very different from what we study here. 

Remark 1.1.5. Note that a type descent is actually an ascent. 

Remark 1.1.6. Often, the last letter (or, the last position) of a permutation is 
considered to be a descent but not here. 

Example 1.1.7. When r = 3, n = 6, 



TT = 544453222335611166 
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is a 3-permutation of {1,2,..., 6}. Then 

S{n) =5(544453222335)5(111)6 
=5(444)5(322233)516 
=45(222)3516 
=423516. 

The sets of type i descents (i = 0, . . . , 3) of vr are ir^^^ = {6}, ir^^^ = {3, 5, 6}, n^'^^ = {5} 
and TT^^) = 0. 

The r-permutations can be represented as (r + l)-ary decreasing trees. If we have 
an r-permutation vr of [n] and vr = ai?7,a2?7. ■ ■ -nar+i, to get the tree representation of 
71, we set n to be the root of the tree and recursively set the tree representation of 
{i = 1, 2, . . . , r + 1) to be the ith child of the root. If a, is nonempty, then vr has an ith 
child and n is a type i — 1 descent. Therefore it is clear that each ith child corresponds 
to a type i — 1 descent. For example, the 3-permutation above can be represented as 



in Figure 1.1 



There are three kinds of traversals of a tree: preorder, inorder, and postorder [|23 



p. 243; |T^, p. 315]. In a preorder traversal of a tree, the root is visited first and 
then the subtrees rooted at its children are traversed recursively. A postorder traversal 
recursively traverses the subtrees rooted at the children of the root first, and then visits 
the root. An inorder traversal, in our case, recursively traverses the subtree rooted at 
the first child of the root, then visits the root, then recursively traverses the subtree 
rooted at the second child of the root, then visits the root, and so on. Hence, if we 
read the tree in Figure [I.l| in inorder, we get back the original r-permutation vr. If we 
read it in postorder, we get an ordinary permutation and it corresponds to 5(7r). 
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Figure 1.1. Tree representation of tt 
Because any (r + l)-ary tree on n vertices has n — 1 children, we have 

Lemma 1.1.8. Every r -permutation of [n] has a total of n — 1 descents of types 
0,1,. ..,r. 



1.2. Stack-sortable r-permutations 

Now let us first consider the enumeration of 1-stack-sortable r-permutations (which 
we call here stack-sortable r-permutations) with all types of descents. 

We weight any type i descent by Xi. Therefore the weight of the r-permutation tt 
in Example p..l.7| is Xoxlx2- Because of Lemma |1.1.8| , we do not need a parameter to 
keep track of the number of different letters in an r-permutation. 

For any unlabeled (r -|- l)-ary tree, there is one and only one way to label the 
nodes to make it an (r -|- l)-ary decreasing tree such that if we read it in postorder, we 
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get an identity ordinary permutation. That gives a bijection between stack-sortable 
r-permutations and unlabeled (r + l)-ary trees. Therefore, if we weight an unlabeled 
(r + l)-ary tree by weighting any zth child by Xi^i, then the total weight of stack- 
sortable r-permutations is the total weight of unlabeled (r + l)-ary trees. Let A{x) be 
the weight of all stack-sortable r-permutations. Then we have 



Aix) =l[il+XiA{x)) 



1=0 



where x = {xq 



To solve this functional equation, we use Lagrange inversion |[10| , p. 21 



Lemma 1.2.1. (Multivariable Lagrange Inversion) Let /(A) G IR[[A]] and 0i(A), . . . , 
0m(A) G M[[A]] where A = (Ai, . . . , Am). Suppose that Wi = ti(f)i{w) for i = 1, . . . ,m, 
where w = {wi, . . .,Wm)- Let = (0i, . . . ,0^), t = {ti, . . . ,tm) and k = {ki, . . . ,km)- 
Then 



_ Xj d(f)i{X) 
0.(A) dX, 



where [A'^]/(A) is the coefficient of A^^ ■ ■ ■ X^ of the formal power series /(A). 

In our case, we let Wi = XiA{x) and 0i(A) = 11^=0(1 + -^i) for i = 0, . . . , r. Then 
Wi satisfy the condition that Wi = Xi(f)i{w). Also let /(A) = Hj^ol-'^ + ^j)- Then 
f{w) = A{x). Using multivariable Lagrange inversion, we get 



A{x) =/H 



A:o + ---+fcr+l 



k I \j=0 

Now we introduce some notation. Let 



l + A, 



;i.2.i) 



(ik{x) = 

0<n<i2<---<jfc<r- 



X/ >^ . , , , 'Y' • 
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be the A;th elementary symmetric function of x = {xq, Xi, . . . , Xr). Define 



(1.2.2) 



E{x, u) —E{xo, Xi, . . . , Xr] u) 

r 

^Y[{l+XiU) 

i=0 
r 



i=0 



and let 



:i.2.3) 



E{x) = E{x,l). 



Also we introduce the following lemma to evaluate the determinant. 

Lemma 1.2.2. Let M{x) — \Sij + {Sij — i)xj\, where < i, j < r, be the determi- 
nant of the (r + 1) X (r + 1) matrix. Then 



M{x)^E{x)ll-J2j 



i=0 



+ Xi 



Proof. 



M{x) 



1 

-Xq 



-Xi -X2 
1 -X2 



-Xq -Xi -X2 
-Xq —Xi —X2 



-Xr-1 1 
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Subtracting the last row from every other row, we get 
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M{x) 



1+Xo 
1 + xi 

1 + X2 





-Xo -Xi -X2 



-{1+Xr) 

-{1+Xr) 

-{1+Xr) 

1 + Xr-l -{1 + Xr) 
—Xr-1 1 



If we let x' = {xi,X2, . . . , Xr), then by expanding the first column, we get 



M{x) =(1 + Xo) 



1 + xi 

1 + X2 



-{1 + Xr 
-{1 + Xr 



-l)''xo 



-Xi -X2 



1 + xi 



-{1+Xr) 
-{1+Xr) 



••• 1 + Xr-l 1 
r 

=(1 + Xo)M{x') - (-l)'-xo(-l)'- 1[{1 + Xi) 

i 

r 

=(1 + Xo)M{x') - Xo YI{1 + Xi). 



1=1 



i=l 



Now we use induction. It is easy to see that the lemma is true when r — 0. Suppose 
the lemma is true for M{x'), i.e.. 



M{x') = E{x') 1 - E I 



+ Xi 
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Then 



Mix) =(1 + Xo)Eix') Yf^, - ^0 + 

\ i=l */ i=l 

=E{x) f 1 - V 7^ I - E{x)-^ 



1=1 

r 



- + Xi 



The lemma is proved. □ 



With Lemma [1.2.2| , it follows that 

{/ J. \ fcoH hfcr + l / r 

\i=0 / \i=0 ' 

^E^4E(:)-("r)-(:)-^n(: 



n — A;,' 



k 



n 



A; i=0 ^ 

where n = 1 + kQ + ■ ■ ■ + k^- Therefore, 

Theorem 1.2.3. The number of stack- sortahle r -permutations with ki descents of 
type i is 

n 11 \k. 

where n = \ + kn + ■ ■ ■ + kr. 
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Remark 1.2.4. This functional equation can also be solved by one variable La- 
grange inversion by introducing a new variable, i.e., letting 



A{z) = zY[{l+XiA{z)). 



1=0 

Remark 1.2.5. These numbers also come up in counting noncrossing partitions 



21, 20 



Remark 1.2.6. When r = 1, these numbers are Narayana numbers P^l O] 



CHAPTER 2 



Two-stack-sortable r-permutations 

2.1. The functional equations 

Now let us consider the case of two-stack-sortable r-permutations. An r-permutation 
TT of [n] is two-stack-sortable if S'^{tt) is an identity permutation. Let tt = aina2n • ■ ■ nar+i 
where n is the largest element in tt. Then 

S{7r) —S{ai)S{a2) ■ ■ ■ S{ar+i)n 
S'^(7r) —S{S{ai)S{a2) ■ ■ ■ S{ar-\-i))n 

Denote the identity permutation by /. (Here we abuse the notation by letting / 
be the identity permutation of [n] for any positive integer n.) One can notice that if 
S'^{Tr) = I and if m is the largest number in aia2 ■ ■ ■ ctr+i, then m can occur in only 
one of the ctj because if m appeared in two of the ctj, the r-permutation condition for 
TT would be violated. Thus (ai, a2, ■ ■ ■ , ocr+i) is an (r -|- l)-tuples of r-permutations 
satisfying the condition that 

S{S{ai)S{a2) ■ ■ ■S{ar+i)) = /. 
This suggests that we study the subject defined as follows. 

Definition 2.1.1. (ai, a2, . . . , ak) is a fc-tuple r-permutation of [n] if aia2 ■ ■ - otk 
is an r-permutation of [n\ and any letter appears in only one of the ctj. Also we call 
Qij the ith component of the /c-tuple r-permutation. 

15 
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Figure 2.1. Forest representation of vr 

We call a fc-tuple r-permutation (ai, a2, • • • , a two-stack-sortable A;-tuple r- 
permutation if it satisfies the condition that 

S{S{ai)S{a2) ■ ■ -Slak)) = I. 

From the definition, we know that each component of a /c-tuple r-permutation 
is itself an r-permutation so that we can interpret a fc-tuple r-permutation of [n] as 
a forest of (r + l)-ary decreasing trees. The set of the descents of type i of /c-tuple 
r-permutations vr = (ai, . . . , a^) is the union of the sets of descents of type i of aj 
{j = l,...k), i.e. 

For example, let vr = {ai,a2) where ai = 553111335 and a2 = 442224776667. 
Then (ai, 0^2) is a 2-tuple 3-permutation of {1, 2, . . . , 7} and it can be represented as 
in Figure |2T^. Since 

S{S{ai)S{a2)) =5(1352467) 
= 1324567, 

TT is not two-stack-sortable. The sets of descents of vr are 7r'-°^ = {7}, vr^^-* = {3}, 
vr(2) = {4,5,7}, 7r(3) = 0. 
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(k) 

Let gn be the sum of the weights of two-stack-sortable /c-tuple r-permutations 
of n different letters such that every component is nonempty. We need every com- 
ponent to be nonempty so that we can keep track of every type of descent. Suppose 
(ai, 0^2, . . . , Ofc) is a two-stack-sortable A;-tuple r-permutation of [n] in which every com- 
ponent is nonempty and the largest element n appears in a/. Let ai = [3in[32n ■ ■ ■ nPr+i- 
Then 



(2.1.1) 

/ =S{S{ai)S{a2) ■ ■ -Siak)) 
=S{S{ai)S{a2) ■ ■ ■ S{ai.i)S{pi)S{p2) ■ ■ ■ S{pr+i) n S{ai+i) ■ ■ ■S'(afc)) 
=S{S{a,)S{a2) ■ ■ ■ S{ai.i)S{Pi)S{(32) ■ ■ ■ S{Pr+i)) S{S{ai+,) ■ ■ ■ S{ak)) n. 

Therefore, (ai, . . . , ai-i, Pi, ... , Pr+i) and («;+!, . . . , Ofc) are both two-stack-sortable 
tuples of r-permutations. 

Let us first consider the case when / = 1. By we have 

/ = SiSiPi)SiP2) ■ ■ ■ SiPr+i)) S{Sia2) ■ ■ ■ Siak)) n. 

Suppose there are n — i letters (including n) in ai. Then the weight of (0:2, as, . . . , a^) 
is a term of (7-^ ^\ For {(3i, . . . , (3r+i), if all (3j are nonempty, then the weight of 
(/?!,..., (3r+i) is a term of Qn-i-i- But by the definition of the descents of r-permutations, 
there is a descent of each type in ai for every appearance of n. So the weight of ai is 
times the weight of . . . Therefore the sum of the weights of all 

such «! is er+i{x)gn-i-i- If only one of the (3j is empty, then the weight of . . . , /?r+i) 
is a term of g^^li_^. Again, by the definition of the descents of r-permutations, there 
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are r — 1 descents of different types for r — 1 appearances of n. So, if is empty but 
j3j is not empty for j ^ Jq, then the weight of ai is XqXi ■ ■ ■ Xj^ ■ ■ ■ Xr times the weight 
of (Pi,. . . ,Pr+i)- Therefore the sum of the weights of all such cti is er(x)g^}_i_^. In 
general, if there are m of the Pj that are nonempty, then the sum of the weights of all 
such ai is em{x)g^^_^. Therefore, the sum of weights of all the /c-tuple r-permutation 
(ai, a2, . . . , ctfe) in which n is in ai is 

n— 1 

E 9t%Sl.-^ + ^2i-.ei(x) + • • • + ££.e.-,,(x)], 

i=0 

For the same reason, when n is in a2, we have 

I = S{S{ai)S{Pi)S{P2) ■ ■ ■ S{Pr+i)) S{S{as) ■ ■ ■ S{ak)) n. 

Suppose there are n — i letters in {ai,a2)- Then the sum of the weights of all the 
/c-tuple r-permutation (ai, a2, ■ ■ ■ , Oik) in which n is in a2 is 

n-1 

and so on. Thus we have the crucial recursive formula 

n— 1 k—1 r+1 

(2.1.2) 9^:^ 

i=0 1=0 j=0 

with the initial conditions that g^^^ = Sok and gn^ = Sno- 
Now let 

(2.1.3) /i'^ = Eyi'+^'^e,(x). 

j=0 

Then 

n— 1 n— 1 



(2. 1.4) ) = ft." + /i-"-'*"" + ■ • ■ + E /f 



(0) Jk-1) 

J: 

i=0 i=0 
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(k) 

Following the analysis above one can easily see that /„ is the sum of the weights 
of {k + l)-tuple r-permutations (01,0:2, . . . , Ofc+i) of [n + 1] such that the Oj are all 
nonempty, S{S{ai)S{a2) ■ ■ ■ S{ak+i)) = I and the largest letter only appears in a^+i- 
Also let 



00 00 



(2.1.5) fk=J2fn\ F = Y,fkz' 



n=0 fc=0 

00 00 



(2.1.6) 9k=Y^9^::\ G = j2 

n=0 k=0 

Then it follows that 

9o =1, 
/o =gi, 

(2.1.7) fk =gk + gk+iei{x) H h gr+k+ier+i{x) , 

(2.1.8) gk =fk-igo + fk-2gi H h 



gkz''. 



and from ( p.l.8|) , we get 



(2.1.9) G = l + zFG. 



Notice that if we let r go to infinity, we can interpret fc-tuple r-permutations in terms 
of forests of decreasing trees and equations ( |2.1.7| ) to ( ^.1.91 ) are still well defined. 
Therefore, we have the functional equations as follows. 
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Theorem 2.1.2. Let F and G be defined as in ( ^.l.d^ j and Then F and G 

satisfy the functional equations 



(2.1.10) 



f2.1.111 



fk = Ygk+jej{x) for k>0, 

j=0 

G=l+ zFG 



and F and G are uniquely determined as power series by these functional equations. 



Remark 2.1.3. From ( |2.1.4| ), we see that gli^ are uniquely determined by //"'■' and 
(yfp^ where i < n. From (|2.1.3|) , we see that fn^ are uniquely determined by gn^. 



Therefore, given gj^^ = 1 and g^''^ = when > 1, all gli^ and fn^^ are uniquely 



determined by (|2.1.3|) and (^.1.41). Hence, F and G are uniquely determined by the 



functional equations in Theorem |2.1.2 . 



Remark 2.1.4. Notice that gi = /o is the generating function of two-stack-sortable 
r-p ermut at ions . 



Remark 2.1.5. Notice that in the ring Q[[z 



G-Elx,- 



XQ XX X2 

■• z ' z ' z ' 



\ CXJ CXJ 

' k=—oo j=0 



Therefore, by ( p.l.lO|) , F — G ■ E (^x,^) has only negative powers of z. This is the key 
to solving the functional equations ( p.l.lO[ ) and (|2.1.11| ). 



2.2. The solution to the functional equations 

Let y = {yo,yi,y2, ■■), where the j/j are uniquely determined as formal power series 
in the Xi by 



2.2. THE SOLUTION TO THE FUNCTIONAL EQUATIONS 

Let t = zE^{y) where E{y) is defined as in ( JLT^ ) and ( [1X31 ). Let c{t) = 
be the generating function of the Catalan numbers. Then 



1 - Vl 
2t 
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Theorem 2.2.1. The solution to the functional equations in Theorem |^. i.^ is: 



(2.2.1) 
(2.2.2) 



G 



cm{y) 
'E{y,c{t)y 



t 



c{t) 



The proof of the theorem consists of the following lemmas. 



Lemma 2.2.2. In the ring Q[[z 



Proof. First, we notice that E ( x,-] G Q\\z, — , — , . 

\ z J z z 

' •••]]• ^-Iso easy to verify that Q[[z, — , — , 

Therefore, since c(t) satisfies tc^{t) — c{t) + 1 = 0, 



' t' t' 



[I + c{t)yi) 1 + 



tc{t) 



= 1 + 



Vi + 1/i 



=1 + ^. 

z 
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Hence, 
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°° / I 2 \ 

OO / s 

-n(i-(«(i+s|)) 

=Bfe.c(t))B(,.^). 



Lemma 2.2.3. Le^ 



A _ c{t)E{y) 

E{y,c{t)y 

Then G and F satisfy 

(7 = 1 + zFG. 



Proof. 



_ zE\y) f _ E{y,cit)) \ 
t V c{t)E{y)J 
E{y,c{t)) 



c{t)E{y) 
1 

d 



□ 



2.2. THE SOLUTION TO THE FUNCTIONAL EQUATIONS 

Lemma 2.2.4. In the ring Q[[z, f,f,f,... ]], define 

F- ^ G ■ E (^x,^^ - F. 
Then F~ has only negative powers of z. 
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Proof. 



Since 
(2.2.3) 



E{y,c{t)) \' z) i V 
c{t)E{y) 



E(y,c(t)) 

E{y) 



E{y) 



, E{y,c{t)) 



tc{t)E y, 



tc{t) J c{t) 



-E{y) 



Y,HtT-' + t'-''c{tf-^-l\ eM 



n=0 

oo 



n=0 



en(y)- 



itcit)r-' + city 



n— 1 



k even 
k even 



is a polynomial of degree less than or equal to (n — l)/2, F has only negative powers 
of t and thus has only negative powers of 2;. □ 
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Lemma 2.2.5. Let G and F he defined as above. Define 



fk = [z']F, 



9k = 



Then and fk satisfy the functional equation that 



fk = ^gk+jej{x) for k>0. 

j=0 

Proof. From the definition of fk and cjk, we know tliat 

oo 



k=0 

oo 



gkz 



Then 



k=0 



[z'] ^G-E{x,^^-F^ = J29k+Mx) - fk. 



By Lemma 2.2.4| , this sum is zero when k > 0. 
The lemma is proved. 



□ 



Now Theorem [^.2.11 follows from Lemma [^.2.31 and Lemma 12.2.5 



2.3. The number of two-stack-sortable r-permutations with descents 

We know that the generating function for two-stack-sortable r-permutations is 

9i =/o 
= [z°]F 



I n=0 
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1 — n fn + 2i^ 



1^ n=0 i=0 ^ 

oo 

=E(|/)^(l-n)e„(y) 



Je„(|/)f 



n=0 

where 



We can now evaluate the number of two-stack-sortable r-permutations with the help 
of multivariable Lagrange inversion. 

Theorem 2.3.1. The number of two-stack-sortable r-permutations with ki descents 
of type i is 

where n = 1 + /cq + /ci + • • • + fcr- 

First, we introduce the following identity: 
Lemma 2.3.2. 

r+l / r 



i=Q \ i=Q 

Proof. Since ei{y) is a homogeneous polynomial in yo, yi, . . . , of degree i, 

yjdciiy) 



te. 



Also notice that 



'^'^^ ay, 



dE{y) _ E{y) 
dyj 1 + y/ 
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Thus 

r+l r+1 



5^(1 - z)e,(i/) =E(y)-^^e,(2/) 

i=0 1=0 

1=0 j=0 '^•^ 
j=0 i=0 

r 



Therefore, 

oo 

(2.3.2) g,=E{y)J2i'^-n)eM 



n=0 

r 

n 



If for any y = {yo, yi, . . . , yr), we define 



,=0 - ■ 



A{y,u) =A{yo,...,yr;u) 



1=0 ^* 



and let 



A{y)=A{y,l), 



□ 



then gi = E\y)A{y). 
The proof of Theorem 



2.3.1 



now consists of following lemmas. 
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Notice that for any formal power series /(x) = J2k ^kx'^ ^ Qlt^^o? xi, . . . , Xr\\ (where 
k = (fco, . . . ,kr) and x = (xq, . . . , Xr)), if we define operators D^, I^: Q[[xo, Xi, . . . , Xr]] — 
Q[[xo,Xi, . . .,Xr]] by 

i=0 



dxi 



then {Dx + Ix){f{x)) = nakX^, where n = ko + ■ ■ ■ + kr + I. Therefore, to prove 

1 -A- n f2n — \ — k,;^ 
\9i = — I I 

r) ^ J- J- n — 

we only need to prove that 



i=0 



[x'=](D.+j.)v=n 

4 = 



^ n f2n — 1 — k. 



Tl \ k' 



Let = ^— . Then 



Lemma 2.3.3. 



l + 2y, A{y,2) 



Proof. Take the logarithm of both sides of Xi = — — / . We get 
(2.3.3) Inx, = Iny, + ln(l + y,) - 2ln E{y). 



Then by differentiating both sides of |2.3.3| with respect to Xj, we get 



(2.3.4) — = — -7- r^~;^ ! II J = ^) 

Xi yi[l + yi)dxi E[y) dxi 

(2.3.5) ,.2±^J^-J-^, if,^. 

yi[l + yi)dxj E{y) dxj 
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Now multiplying Xj by both (2.3.4) and ( |2.3.5 ) and summing on j, we get 



^ _ l + 2yi Xjdyi _ 2 XjdE{y) 



(2.3.6) =4tT^"^ - 

y,{l + yi) E{y) 

On the other hand, 

dE{y) 



DAEiy))=J2 



Xj- 



dxj 

3=0 ^ 



dE{y) % 



i=0 ^ + ^' j=0 

(2.3.7) =^(^)Er^ 



-Mi 



By solving (|2.3.6| ) and (p.3.7|) for Wj, we get 

yiil + yi) 1 



Ui 



l + 2yi A{y,2y 

The lemma is proved. □ 

Using the lemma above, the proofs of the following lemmas are just a matter of 
simple computations. 



Lemma 2.3.4. 



Proof. 



dxj 

3=0 ^ 
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?2i 



dE\y) dy, 

j=0 i=0 



3 Qy, Q^, 



E 



dEHy) 



_^ 2E\y)y,{l + y,) 1 
^ l + y^ 1 + 2?/, A{y,2) 



i=0 
?2 



My. 2) 1^ 1 + 22/. 



^(2/,2) 

The lemma is proved. □ 



Lemma 2.3.5. 



Proof. 



A{y, 2) ' 



,=0 



Xj- 



r r 
j=0 i=0 



dA{y) dy,, 
dyi dxj 



E 

i=0 



dA{y) 
dyi 



1 yi{l + yi) 1 



E (i + ^,)2 i + 2|/, A{y,2) 



y,2)j^[l + yi l + 2yj 
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{A{y,2)-A{y)) 



Lemma 2.3.6. 

{D^ + h){g^) = E\y) 

Proof. 

{a, + h){g^) ={D. + h){E\y)A{y)) 

=E\y)D.M{y)) + A{y)D^{E\y)) + E\y)A{y) 



Now from Lemma [2.3.4] to Lemma |2.3.6| , we see that 

E\y) 



Lemma 2.3.7. 



\X' 



A{y,2] 

.1 E\y) _ 
^A{y,2) 



(D^ + I^Yig,). 



n 

i=0 



n f2n — 1 — k, 



n — k 



ki 



where n = kn + ■ ■ ■ + + 1. 



Proof. Using multivariable Lagrange inversion (Lemma |1.2.1| ), we set 

, E\y) 
<Pi = ^ — • 
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Thus Ui = XiCpiiy). Using Lemma |1.2.2| to evaluate the determinant, we have 

Vj d(f)i{y) 
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\x 



.1 E'iy) 



(l>iiy) dyj 



^\A{y,2f^y> E{y) 



{ E\y) ,,, ,E(y,2)A(y,2) 



E{y) 



-[y']{ fr',^l E{yX 



n 

i 

n 



2n - 1 - A;, 



+ 2 



2n — 1 — ki 
ki-l 



n j 2n — 1 — ki 

Til \ kj 



Therefore, Theorem 2.3.1 is proved. 



Remark 2.3.8. When r = 1, we have kQ + ki = n — 1. Then the number is 
1 



n 



'2n-l- ko\ n f2n-l- ki 
n — k^y ko ) n — ki\ ki 

{n + ki)\ {2n - 1 - ki] 



{ki + 1)! (2fci + 1)! {n - ki)\ {2n - 1 - 2ki)\ 
which was found and proved by Bona f^, Bousquet-Melou [§, and Travis 

2.4. The number of two-stack-sortable r-permutations 



□ 



The following theorem is a special case of Theorem |2.3.1| when we do not keep track 
of descents. 

Theorem 2.4.1. The number of two-stack-sortable r-permutations of [n] is: 

((2r + l)n)! 



2(r + l) 



n\ (2rn + 2)!' 
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First proof: 

We know that the generating function for two-stack-sortable r-permutations is given 
by (|2.3.2| ). Setting yi = y for i = 0, ... ,r, which is equivalent to setting Xi = x for 



i = 0, . . . , r, we get that 

y 



X 



2r+l 



and 



(2.4.1) 



<r + l)(l+t/)2'-+i-r(l+t/)2'^+2 

il + yy^+\l - ry) 
y{l - ry) 



X 



Using Lagrange inversion [|T0|, p. 17; Q , we set (j){y) = (1 + ?/)^''+^. Then y satisfies 
the condition that y = x(f){y). Therefore, 



[x']gi =[x''+^]xgi 



:^[/]{(l-2ry)(l + y)(^^+^)(^+i)} 



1 f /(2r + l)(A; + l)^^ /(2r + 1)(A: + T 



k+1 {\ k J \ k-1 

..J, I n ((2r + l)(fc + l))! 
" ^ ^(A; + l)!(2r(A; + l) + 2)!' 



Setting n = /c + 1, then the theorem is proved. 
Second proof: 
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From Theorem ^.3.11 , we know that the total number of two-stack-sort able permu- 
tations is 

^ n f2n — \ — kj 



- y w 

'n? ^ ^ ^ n — ki \ ki 

fcoH \-kr=n—l i=0 



Also we know that 



oo 



for any integer n p. 154]. Therefore, 



j=0 



n f2n — 1 — k. 



Hence, 



n /2n — I — kj 
^ n — ki\ kj 



fcoH hfcr=ra— 1 i=0 



1 {c^^i-t)^-' 



^ , ^ 2n{r + 1) (23 - 2n{r + 1)\ . 
n^^' J 2n(r + 1) - 2j1 j / 

. 1 2n(r + l) /2(n - 1) - 2n(r + 1) 

"n2 2n(r + l)-2(ra-l)^ ^ V 
r + 1 /(2r+l)n 



n(nr + 1) V ^ — 1 

=2(r + l)i^^!:±lMi 
^ ^n!(2rn + 2)!' 

The theorem is proved. □ 
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Remark 2.4.2. When r = 1, this number is 

^ (3n)! _ 2(3n)! 
n\{2n + 2)\ (n + 1)! (2n + 1)! ' 

which was conjectured by West |2^, ^ and first proved by Zeilberger 



CHAPTER 3 



Parallel results 



3.1. Another approach 



In the previous chapter, we counted the number of two-stack-sortable r-permutations 
with all types of descents. Here we approach a special case of the problem from an- 
other angle. We now count the number of two-stack-sortable r-permutations with only 
two basic parameters, the number of different letters and the number of components. 
We use the indeterminates x and z to count these two parameters respectively. The 
difference is, that any component is allowed to be empty. 

Therefore, using the same decomposition method as in Chapter 2, if we let pn 
be the weight of all A;-tuple r-permutations (ai, . . . ,ak) of [n] where each of {i 
1, . . . , fc) is allowed to be empty, and let qn^ = pn^''^^\ 



(3.1.1) 
(3.1.2) 



n=0 

oo 



n=0 



k=0 

oo 



fc=0 



then analogous to Theorem |2.1.2| , we have that the formal power series P and Q satisfy 
the functional equations 



(3.1.3) 
(3.1.4) 



Qk —Pr+k+1, 
1 



P 



1-Z 
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Since 



Q 



i=0 



we have that 



Theorem 3.1.1. The generating function P of two- stack- sortable k-tuple r -permutations 
satisfies 



P 



(3.1.5) 

and P is uniquely determined as a power series by this functional equation. 



Now, compared to the functional equations (|2.1.10| ) and ( p.l.ll|) , this one is struc- 
turally simpler. We will also see that ( |3.1.5| ) can be easily derived from the functional 
equations ( ^TIoD and ( ^TTTl) . 

If we set Xi = X m. functional equations (|2.1.10|) and ( p.l.ll|) , then they become 

r+l 

(3.1.6) 



(3.1.7) 



fk=^{^ ^.^ ] Qk+jX^ for k>0, 
G =l + zFG. 



Notice that pk is the weight of of all fc-tuple r-permutations where any component 
is allowed to be empty and Qk is the weight of of all /c-tuple r-permutations where no 
component is allowed to be empty. Therefore, 

k 



(3.1.8) 

which is equivalent to 



Pk 



E 

i=0 
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Setting Xi = x, which is equivalent to setting i/i = y, in the solution of functional 
equations (Theorem |2.2.1|) gives the solution to functional equation ( p.l.6|) and (|3.1.71 ), 

c(t)(i+yr+^ 

{l + yc{t)y+^' 

where y is uniquely determined as formal power series in x by 

y 



X 



:i + y) 



2r+l 



and t = z{l + 1/)^*^''+^''. Therefore, 



Theorem 3.1.2. The solution to function equation (^.1.^ is: 

t \c{t){l + yy+^ 



P=\l 



y{l + y)J (l + i/c(t)y+i' 
where y is uniquely determined as a formal power series in x by 

y 



X 



2r+l 



and t = y{l + y) 



l-z 



Corollary 3.1.3. pk is a polynomial in y with degree no greater than 2k. 
Proof. 

x'^gk =x''[z'']G 

G 

c(t)(i+yr+i 



l+yc{t)y+^ 
c{xt){l + yY+^ 



'l+yc{xt)Y+^ 

oo 



1=0 



r + 1 
i 



y\c{xt)) 



i+l 
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^1(1+1/) 



c r')4 c^^r >'(^ -^)- } 

The sum is a hypergeometric series that can be transformed by Euler's transformation 
0. Thus, 

Therefore, x^gk is a polynomial in y with degree no greater than 2k. Thus pk is a 
polynomial in y with degree no greater than 2k because pk = X]i=o (i)'''*^'* (■^'l- 



□ 

In particular. 

Corollary 3.1.4. pi is the generating function of two-stack-sortable r-permutations, 
and 

p^ = l + y- ry'^. 

Proof. Since pi = 1 + xgi (Eq. ( pLSD ) and = ^'^^ ~ (Eq. (|2XT| )), 

Pi =1 + xgi 
=1 + y~ry'^. 



□ 
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3.2. Connection to Zeilberger's functional equation 

Zeilberger first proved West's conjecture that the number of two-stack-sortable 

2(3n)\ 

permutations of length n is -, r—, rr PStl . He used a factorization similar to 

^ ^ (n + l)!(2n + l)! ^ 



ours and the functional equation he got is equivalent to our functional equation ( ^.1.5 ) 
in the case of r = 1. 



In Zeilberger's paper |2^, he defined i{TT) (where vr is any permutation of {1, 2, ... , n}) 
to be the largest integer i such that the subsequence of the 'big i': {n—i+1, . . . , n—1, n} 
are in decreasing order, defined W^^^ to be the set of all permutations (of any length) vr 
such that z(7r) = i, and let W-"^ to be the set of all permutations tt such that i(7r) > i. 
Also he defined W^^^ (x) to be the formal power series that equals to the sum of all the 
weights of elements of W^^\ and W-'^{x) to be the formal power series that equals to 
the sum of all the weights of elements of W-\ and he defined 

oo 

i=0 

oo 

$(a;,t) ■.= J2W-'{x)t\ 

i=0 

Then, he got 

[6.Z.L) n^,t) - ^—-^ + ^^—-y^ . 

Now noticing that $(x, 1) = <i>(x, 0) and 

Equation ( |3.2.1|) is equivalent to 



- ^ 1 ^ (1 + xt^x, t))($(x, t) - $(x, 0)) 



xt t 



40 3. PARALLEL RESULTS 

Now it is easy to check that P{x, z) = 1 + (^x, — j . Therefore, the functional equation 



(|3.2.1|) is equivalent to (|3.1.5| ) in the case of r = 1. 



The combinatorial connection between the set W-^ and the set of fc-tuple permu- 
tations was clearly stated in Zeilberger's paper ||28|. For a typical member vr of W-\ 
if its length is n, then it has the form 

TT = aonai{n — 1) ■ ■ ■ {n — i + l)ai, 

where ao, ■ ■ ■ ,ai are (possibly empty) permutations of disjoint smaller sets, the union 
of whose underlying sets is {1,2, . . . ,n — i}. Now, by iterating the definition of the 
stack sorting operation S, 

S{'7i) = S{ao)S{ai) ■ ■ ■ S{ai){n — i + l){n — i + 2) ■ ■ ■ n, 

so that, 

S\n) = S{S{ao)S{ai)---S{ai)){n-i + l){n-i + 2)---n. 

It follows that there is a 1-1 correspondence between the elements of W-^ and {i + 1)- 
tuple permutations ao, ai, . . . , a^, such that S{S{ao) ■ ■ ■ S{ai)) = I, and the underlying 
sets of the a's are disjoint and their union is {1, 2, . . . , n — i}. 



CHAPTER 4 



Further results 
4.1. A generalization of the stack-sorting operation on r-permutations 

Here we introduce a more general form of the stack-sorting operation on r-permutations. 

Definition 4.1.1. For a given positive number r, let A = (Aq, Ai, . . . , A;) satisfy 
the condition that < Aq < Ai < • • ■ < A; = r . Let vr be an r-permutation of [n]. 
Then we can write vr as vr = aQuain- ■ -nar- The generalized stack-sorting operation 

on r-permutations is defined by 

S'A(7r) = Sx{ao) ■ ■ ■ Sx{axo)nSx{axo+i) ■ ■ ■ Sx{ax^)n ■ ■ ■ nSx{axi_-,+i) ■ ■ ■ Sx{axt)n. 

Notice that Sx{t!') is an (/ + l)-permutation and Sx = S when / = 0, where S is the 
ordinary stack sorting operation on r-permutations (Definition |1.1.2|) . 



It is clear that it is not interesting to consider stack-sortable r-permutation under 
this definition. Also, in order to consider two-stack-sortable r-permutations under this 
definition, we have to consider S{Sx{tt)) instead of Sl{n). 

To enumerate the number of two-stack-sortable r-permutations with descents under 
this definition of stack-sorting operation, we still weight any type i descent by Xi. 
Furthermore, we denote x^^^ = (xq, xi, . . . , xaq) and x^^^ = {xxi_i+i, ■ ■ ■ , xxj for 1 < 
i < I. Also, we use u to keep track of the number of different letters. 
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Now let gn be the sum of the weights of two-stack-sortable A;-tuple r-permutations 
ai, a2, . . . , ak) such that every is nonempty and let 



Ao+l 
j=0 



f^= ^Y.' gi'^-'H^^'^) for l<z</. 

j=0 

Also let 

oo oo 

(4.1.1) /m=E F, = Y,fk, 

n=0 A;=0 

oo oo 



k 

n=0 fc=0 

Then we have the following functional equations by the same reasoning as in section 

H: 



Lemma 4.1.2. Let G and fk^i be defined as above, then 

I 

(4.1.2) G=l + zuGFol[fo,,, 

1=1 

Ao+l 

(4.1.3) /fc,o = 5^^?fc+ie,(x(°)), 

(4.1.4) J2 9k+je,{x^'^) forl<t<l. 

3=0 

If we let A = n!=i /o.j) "we can make the substitution x = uA so that the functional 



equations ( 4.1.2|) to (H:.1.4|) become 



(4.1.5) G=l+zxGF^ 



0, 



Ao+l 

(4.1.6) A,o = 5Z^7fc+,e,(a;(°)). 

j=0 
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By Theorem ^.2.1| , the solution to these functional equations is: 

cmiy) 



G 



E{y,c{t)y 



t V 



E\y) 

t = zxE'^iy). From the solution, we can express gi in terms of x and y, i.e., gi = gi{x, y). 



where y = (yo, • • • , 1/Ao) is uniquely determined by xxj = ^^^^ for < j < Aq, and 



Since x = uA, gi can be expressed in terms of u, A, y, i.e., gi = gi{u, A, y). By equation 

gxi, 



I 



i=l 



= n 5Z 9jiu,A,y)e,{x^'^). 

i=l j=0 

Now we can solve for A from the equation above so that A is expressed in terms of 

u,y,x^^\ !<«</, i.e., A = A(u;y;x^^'>;...;x^^^). Therefore, given gi = gi(u,A,y) 

and Xj = t'/'^l^ < j < Aq, we can solve for gi (for example, using Lagrange 
xE^[y) 

inversion) so that gi = gi{u; x^^^; x^^'^; . . . ; x'-'-'). 
Therefore, 



Theorem 4.1.3. // G and Fi are defined in then G and Fi satisfy the 

following functional equations: 

cmiy) 



G 



E{y,c{t)y 



t V c(t) 

where y = (yo, . . . , yx^) is uniquely determined by xxi = ^ f^'^ for < i < Xq, and 

E\y) 

t = zxE'^{y). In particular, the gi are algebraic. 
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Remark 4.1.4. It is easy to get = 1. Also, similar to Equation ( |2.3.2| ), we can 

get 



gi = xE{y) ^(1 - n)en{y) 



n=0 

in this case. 

For example, the simplest nontrivial case is when / = 1 and A = (1,2). We have 
x^^^ = (xo,Xi), x^^^ = {X2), and the functional equations for this case are: 

G=l + zuGFofo,i 
fk,o =9k + 9k+i{xo + xi) + gk+2XoXi 
fk,i =9k + gk+iX2 



Now we make the substitution x = m/o,i. Then the functional equations above 
become 

G=l + zxGFo 
fk,o =gk + gk+i{xo + xi) + gk+2XoXi 



By Theorem p.l.2| , the solution to these functional equations is: 

cmiy) 



G 



E{y,c{t)y 



where y = {yo,yi), E{y) = (1 + yo){l + yi) and y is uniquely determined by xxi 
for 2 = 0, 1, and t = zxE'^iy). 

E\y) 
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Also, from Remark |4.1.4|, we have 



9o =1, 

oo 

(4.1.7) g^=xE{y)Y,{^-n)eM 

n=0 

=x{i + ?/o)(i + - ym) 

=m/o,i(1 + yo){l + yi){l - ym] 



Therefore, 



/o,i =9q + giX2 

=1 + m/o,i(i + yo)(i + - yoyi)x2- 



By solving for /o,i, we get 



^-u{l + yo){l+yi){l-yoyi)x2' 

So, by CT) , 

_ u{l + yo){l+yi){l-yoyi) 

9i 



l-u{l + yo){l + yi){l - yoyi)x2 



Now, since xxi = f^'^ for z = 0, 1 and x = ufo i, we have 

E^{y) 



yi{l + yi){l - u{l + yo){l + yi){l - yoyi)x2) . . ^ i 

uxi = — for z = 0,1. 

E^[y) 



Since {yo,yi) is uniquely determined by the equation above, we can solve for gi 
terms of u,Xo,Xi and X2 by using Lagrange inversion. 
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We can also approach the problem without counting descents in the same way as 
what we did in Section |3.1j . If we let P be the generating function of two-stack-sortable 
fc-tuple r-permutations under Sx for any k, with u and z keeping track of the number 
of different letters and the number of components, then 

Lemma 4.1.5. P satisfies 



(4.1.8) 




To solve this functional equation, first we make the substitution 



i=l 



Then ( |4.1.8| ) becomes 




From Theorem p.l.2| , we know that 



Theorem 4.1.6. The solution to the functional equation above is 




where y is uniquely determined as a formal power series in x by 



y 



X = 



(l+^)2Ao+l' 



t=a;(l + y)Wi) 



z 



1-z 
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and 



X 

i=l 



Since pk is a polynomial in y (Theorem |3.1.3|) and 

y 



u 



:i+i/)2^"+^nUpA.-A,_,' 

we can solve for in terms of u. 

Again, the simplest nontrivial case is when / = 1 and A = (1, 2), and the functional 
equation for this case is: 

1 ^ P{P-p,-piz) 
+ upi- 



1 — z z 

Now make the substitution x = upi. Then the functional equation above becomes 

1 — z z 

y 

From Corollary 13.1.4 Pi = ^ + y — y'^, where y is uniquely determined by a; = — 

+ 

or 

y 

u 



[l + yf{l+y-y^y 
Now we can use Lagrange inversion [|l^, p. 17; ^ to solve for pi. 

Since y = u{l + y)^{l + y — y'^), 

K]pi =^b"~']^(i + + y - y'T 

n / \ 

i 2{n—{) 



-b"-i](l - 2y){l + yr^ j^^-iy (''\ (1 + y)V 
j=0 j=0 \ / \ J J 
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4.2. A characterization of t-stack-sortable permutations 

It is natural to consider counting fc-stack-sortable permutations for > 2 now. 
But althougli people have been trying, little has been found yet, even for three- stack- 
sortable permutations. 

Similar to West's characterization of two-stack-sortable permutations (which did 



not lead to an enumeration) [^, ^ , we can give a characterization for t-stack-sor table 
permutations. 

For a sequence aia2 ■ ■ ■ of different letters, define (oj, a^) to be an inversion of the 
sequence if 1 < z < j < n and > a^. We also define rank(aj) as follows: rank(aj) = m 
if there are exactly m — 1 letters among oi, 02, . . . , a„ that are smaller than a^. 



Theorem 4.2.1. A permutation vr is t- stack- sortable if and only if it does not con- 
tain a subsequence vr' = 0102 ■ ■ ■ aj+2 which satisfies the following conditions: 

(1) rank(at+2) = 1/ 

(2) rank(at+i) =t + 2; 

(3) For any i and j such that 1 < i < j < t and {ai,aj) is an inversion, there does 
not exist a subsequence n" = C\C2 ■ ■ ■ 0/ vr, where < Ci < C2 < • ■ • < and 
s = t -\- 2 — rank(aj), such that vr" appears between ai and aj in vr. 



To prove this theorem, we need some lemmas from West [26, 27 . 



Lemma 4.2.2. (West) If tt is a permutation of [n] and 1 < a < b < n, and if a 
precedes b in vr, then a precedes b in S{tt). 
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Lemma 4.2.3. (West) If n is a permutation of [n] and 1 < a < b < n, and if b 
precedes a in tt, then b precedes a in S'(7r) if there exists c > b such that b precedes c 
and c precedes a in vr. // there is no such c, then a precedes b in S{7t). 

Lemma 4.2.4. (West) Ifb and a form an inversion in S{n), then there exist c> b 
such that b precedes c and c precedes a in n. 



Now let's prove Theorem [4.2.1 



Proof. Tarjan and West proved that the theorem is correct when t = 1 and t = 2 



2% P6| , p7| . We now prove the theorem by induction. 

Suppose a permutation vr contains a subsequence vr' = aia2 ■ ■ ■ satisfying the 
conditions (1) - (3). Let tti = ^(Tr) and suppose that 6162 ■ ■ -^4+2 is the subsequence 
of TTi that was tt' in tt. 

Since vr' satisfies conditions (1) - (2), by Lemma |4.2.2| , precedes at+i in S'(7r) 



for 1 < i < t; by Lemma |4.2.3| , precedes at+2 in ^'(vr) for 1 < i < t. Thus, we 



get either rank(6t_|.2) = t + 2, rank(6t+i) = 1 or rank(6(_,_2) = 1, rank(6t+i) = t + 2. 
Let Tr[ = bib2 ■ ■ ■ bth+i by supposing rank(6(_|_i) = 1 (If rank(6(+2) = 1, we let vr^ = 
6162 • ■ ■btbt+2)- Therefore, vrj satisfies condition (1) with t replaced by t — 1. 



Let rank(ajQ) = t + 1. Then > Oj for 1 < i <t and i 7^ Iq. By Lemma [4.2.2 
precedes in vri for 1 < i < i^. \{ iq ^ t, then and form an inversion in tt' 
for iq < i < t. Since t + 2 — rank(aj(,) = 1, by condition (3), there is no letter bigger 
than appears between aj^, and in tt. By Lemma [4.2.3| , precedes in tti for 
io < i < t. Therefore, a, precedes in vri for 1 < i < t and i 7^ io- So 64 = a^g and 
rank(6t) = t + 1. Hence 7i[ satisfies condition (2). 

Suppose that n[ does not satisfy condition (3); that is, there exist I < i < j < t — 1 
such that bi > bj, and there exists a subsequence = C1C2 ■ ■ - Cg where bi < Ci < C2 < 
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■ ■ ■ < Cg and s = t + 1 — rank(6j), such that tt'( appears between bi and bj in tti. Then Cg 



and bj form an inversion in tti. By Lemma |4.2.4| , there exists some c > such that 
precedes c and c precedes 6j in tt. This contradicts the fact that tt' satisfies condition 
(3). 

Now, since vr^ satisfies condition (1) - (3), by the induction hypothesis, tti is not 
{t — l)-stack-sortable. So vr is not t-stack-sortable. 

Conversely, we can show that if vr fails to be t-stack-sortable, then it contains a 
subsequence that satisfies the three conditions. 

If vr is not t-stack-sortable, then tti = S'(7r) is not (t — l)-stack-sortable. By the 
induction hypothesis, tti contains a subsequence 7i[ = 6162 • ■ ■ h+i which satisfies the 
following conditions: 

(1) rank(6t+i) = 1; 

(2) rank(6t) = t + 1; 

(3) For any i and j such that l<i<j<t — 1 and bi > bj, there does not exist a 
subsequence vr" = C1C2 ■ ■ - Cg where b^ < ci < C2 < ■ ■ ■ < Cg and s = t + 1 — rank(6j), 
such that vr" appears between bi and bj in tti. 

Notice that 6j and form an inversion in tti for 1 < i < t. By Lemma |4.2.4| , 6j 
precedes bt+i in vr for 1 < i < t. In particular, there exists some a > bt such that bi 
precedes a and a precedes bt+i in vr for 1 < i < t. Suppose that aia2 . . . cit+i is the 
subsequence of tt that gets tranformed to -k[. Then it is clear that at+i = bt+i. Let 
tt' = aia2 . . . ataat+i. Then vr' satisfies conditions (1) and (2). 

If vr' does not satisfy condition (3); that is, there exist 1 < « < j < t such that 
tti > ttj, and there exists a subsequence n" = C1C2 ■ ■ - Cg where < ci < C2 < ■ ■ ■ < 
and s = t + 2 — rank(aj), such that vr" appears between Oj and Oj, then by Lemma 
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4.2.2|, the subsequence C\C2 ■ ■ ■ c^-i appears between and a, in tti. This contradicts 



the fact that vr^ satisfies condition (3). Therefore vr' satisfies condition (3). 

The theorem is proved. □ 

Lemma 4.2.5. (Tarjan) A permutation it is stack- sortahle if and only if tt contains 
no subsequence of type 231. 



Proof. This lemma is the case t = 1 of Theorem [4.2.1 . 

By Theorem [4.2. 1| , a permutation vr is stack-sortable if and only if it does not 
contain a subsequence vr' = 010203, which satisfies the conditions that rank(a3) = 1 
and rank(a2) = 3, which means that 03 < ai < 02. Therefore, tt' is a type 231 
subsequence. □ 

Lemma 4.2.6. (West) A permutation fails to he two-stack-sortable if it contains a 
subsequence of type 234 i or a subsequence of type 324 i which is not part of a subse- 
quence of type 35241- If it contains no such subsequence, n is two-stack-sortable. 



Proof. This lemma is the case t = 2 of Theorem [4.2.1 . 

By Theorem [1.2. 1[ , a permutation vr is two-stack-sortable if and only if it does not 
contain a subsequence tt' = 01020304, which satisfies the conditions that (1) rank(o4) = 
1, (2) rank(o3) = 4 and, (3) oi < 02 or oi > 02. If oi < 02, then vr' is a subsequence of 
type 2341. If oi > 02, then vr' is a subsequence of type 3241. By condition (3), there 
does not exist a c > Oi appearing between oi and 02. If there does exist such a c, 
then when c > 03, 01CO2O3O4 is a subsequence of type 35241; when c < 03, CO2O3O4 is a 
subsequence of 3241, which needs to be considered again in the same way. Therefore, 
these three conditions suggest that vr' is a either type 2341 or type 3241 subsequence. 
If it is a type 3241 subsequence, then it is not part of a subsequence of type 35241. □ 
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To enumerate the three-stack-sortable permutations by the same decomposition we 
used for two-stack-sortable permutations, we consider the following object, called a 
/x-tuple permutation, where fi = {fii, . . . and /ij are positive integers. 

Definition 4.2.7. ((ai, . . . , a^J, (a^^+i, . . . , a^^i+^J, • • • , . . . , a^,+...+/,J) 

is a /x-tuple permutation of [n] if ai ■ ■ -0^^+...+^^. is a permutation of [n]. A yU-tuple 
permutation is three-stack-sortable if 

S{S{S{ai) ■ ■ ■S{a^,))S{S{af,,+i) ■ ■ ■ ^K^+^J) • • • ■ ■ ■ 5(a^i+...+^J)) = /. 

By the same reasoning as in the case of two-stack-sortable permutations, if is 
the generating function for three-stack-sortable /i-tuple permutations, we get 

9(Mi,...,fik~i+'i-,i) + g(pi,...,fik^i-i+l,i)9(pk) + 

i=0 i=0 

It seems that this functional equation is hard to solve. 

4.3. A modification on the functional equations. 

If we replace the elementary symmetric functions ej(x) by complete homogeneous 
symmetric functions hi{x), then the functional equations in Theorem |2 . 1 . 2| will become: 

oo 

fk = ^gk+jhj{x) 

j=0 

G=l + zFG 



i=0 / 



Combinatorially, in terms of the tree representation, instead of having at most one 
child of each type, now we can have any number of children of each type. To solve 



4.3. A MODIFICATION ON THE FUNCTIONAL EQUATIONS. 53 

this, everything else will be the same except one modification on the substitution 
which in this case will be 

where H{ii) is defined analogously to E{ii). Then analogous to the identity in Lemma 
we get 



H[x,-\=H{y,c{t))H(y, ^ 



The solutions to these functional equations will be: 

c{t)H{y) 



G 



H{y,cit)y 



Therefore, 



t V c(t) 

gi =/o 

oo 

=H{y)J2i^-n)K{y) 



n=0 



Again, we use multivariable Lagrange inversion to get the coefficient of x in (71, 
where ) and k = {ko, . . . ,kr) and it is 

1 1^ n f2n + 2ki 

11? n + ki\ ki 

where n = 1 + ko + ki + ■ ■ ■ + kr- 

Remark 4.3.1. If we let 

A(n;A:o,...,A:.) = -ll--^^ J 

fc=0 * ^ ^ 
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and 




then 

A{-n; /co, . . . , K) = (-l)""^5(n; /cq, . . . , K). 
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